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ABSTBAC*** 

Eight groups of ten junior college students vere 
given an experinental training session with nanually sinulated 
teaching nachines, each group being taught with a different node of 
teaching nachine operation. The variables vere student response node, 
size of steps betveen successive items, and branching procedure. A 
vritten test vas given to all students after the training session, 
and again three veeks later. The sane test vas given to a control 
group vhich had no training vith the teaching nachines. The use of 
the sinulated teaching nachines led to a significant degree of 
learning by the subjects. Bultiple choice and branching procedures 
reguired less tine but did not affect test scores significantly. 

Snail iten steps reguired significantly nore training tine, but also 
yielded significantly higher test scores. Significant interaction 
betveen the node of response and branching variables resulted fron a 
high nean criterion score obtained by the constructed 
response— non-branching group. Statistical tables supplenent the 
report. fBot available in hard copy due to marginal legibility of 
original document.! (JT) 
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RESULTS OF UTETIAL EEPEPIIOT IH AUTOMATED TEACHIiaG 



Interest in automated teaching has increased greatly in the past ten years, as 
evidenced by the nuiober of papers on the subject given in recent psychological 
and educational conferences and by the variety of teaching machines now available 
for research or cammerclal purposes. This new interest has been stimulated by 
the critical shortage of skilled teachers throughout the country, and by the 

of many psychologists and educators that automated teaching devices can 
be a valuable tool for teachers in both academic and industrial situations. For 
some learning situations the teeichlng machine may surpass its human counterpart, 
particularly where precise control of the timing and contingencies for student 
reinforcatent is critical. 

A "teaching machine" may be defined generally as a device which has the following 
operational characteristics: 

(1) It presents problem materials (questions) to the student; 

(2) The student is required to respond to the materials by some 
overt behavior; 

(3) The machine provides the student with knowledge of results of his 
behavior, usually immediately following each response. This 
"knowledge of results" consists of tel l ing the student whether 
his response was correct, or of providing the student with 
Infozmatlon from which he can evaluate his own response. 

Within the generic definition given above there are many di men sions along which 
one teaching machine may differ from another. The purpose of the present study 
is to investigate three of these dimensions: 

(1) Student response mode. The relative advantages of the multiple 
choice and the constructed response* modes of operation have been 
debated at length in the literature on teaching machines. S> L. 
Pressey (3) has been a leading proponent of the multiple choice 
mode, while B. F. Skinner (U) has gathered many adherents to the 
position that constructed response items are superior. Response 
iK>de was included in the present study both because of its interest 
as a variable per se and because of the possibility that it would 
interact significantly with the other two variables. 

(2) Size of item step. Most investigators in the field of automat^ ^ 

. thAt optimal learning occurs when items presented 
to the stu^t give the student a large degree of support (1,2,4). 
According to these investigators the items should be worded and 
sequenced in such a way that the student finds it simple to pro- 
ceed from one item to the next, and thus receives a large per cent - 
age of positive reinforcements. In the present study, an attempt 
was made to evaluate experimentally the extent of the purported 
superiority of T step condition. Certain items were eliminated 
from one item set to create another set containing fewer items. 



♦ The term "constructed response" refers here to the procedure in which the student 
must cocplete statements by filling in (constructing) one or more missing words. 
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' Eliminated items covered essentially the same subject material as other 

items in the set and introduced no new concepts. The derived set may be 
operationally defined by this process as having larger step size than 
the original set. 

(3) Item sequence control. The great majority of existing teaching machines 
present subject materi.eJ. items in an essenti€J.ly predetermiiied sequence. 
Relatively little speculation, and even less actual experimentation, 
have been devoted to the possible advantages of a sequence detenuined 
by the behavior of the student. 

It is true that most teaching machines in use today have some lim ited 
flexibility in the sense that a student must answer a question correctly, 
or be shown the correct answer, before the next question is presented. 

Being shown the correct answer or selecting the correct answer after 
several tries does not insure, however, that the student has actually 
acquired the necessary knowledge. If he really does londerstand the 
concept involved, it is probably inefficient to present many additional 
questions covering the same concept. If the student does not understand 
the concept, on the other hand, he should not be presented with more 
complex questions based upon the original concept, since he is not 
properly prepared to answer such questions. 

A truly adaptable teaching machine should be capable of discriminating 
critical aspects of the student's response behavior, and selecting 
( subsequent instructional items on the beisls of this behavior ^ Other 

variables which a teaching machine might modify in response to student 
behavior include the amount and type of knowledge of results presented ^ 

to the student, and the number of "cues" given to help the student 
answer the problem materials. Such a teaching machine would act much 
more like a human tutor than do most machines available today. 

In the present study a simple type of sequence flexibility, which will 
be called "forward branching," or merely "branching," was compared 
with the conventional predetermined sequence. The branching procedure 
(described in more detail in the Method section) was Intended to allow 
a student who has already learned a concept to skip over certain other 
items covering this same concept. It was hypothesized that students 
trained with the branching procedure would learn as well as students 
trained without branching, and would learn in a shorter time due to 
the omission of some of the items. 
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The ejqperlBent consisted of three phases: (l) A training session for each of dO suhjecti 

vlth a "beaching machine" operating under one of the el^t different teaching procedures 
being compared; (2) A criterion quiz given to each subject Immedlatelj following his 
training session^ to determine how much the subject learned In the training session; and 
( 3 ) The sane criterion quiz, given approximately two weeks after the training sessions 
all students In the school classes from which the eagperlaental subjects were drawn. 
This last phase was Intended to provide a measure of retention In the subjects, as well 
as a comparison of subjects with other students who have not had the experimental 
training session. 

Althou^ the training session for any single subject reqiolred less than two hours, the 
sessions for all subjects extended over a period of one week. During this weak the 
training sessions replaced regular classroom studies for the e^qperimental subjects, 
while the oontrol group continued to receive Its noxmal Instruction. Subject material 
for the classroom Instruction during this week, and up to the final administration of 
the criterion test, was not closely related to the material tauipit with the teac h i n g 
Machines * 

Six identical sets of equipment were used in the experimental training sessions. The 
sets actually constituted simulated teaching machines, since for purposes of oonran- 
lence human experimenters were used^ In place of automatic control meehaniam. Sack set 
consisted of a wooden screen, a nu]dl>er of pushbuttons and U^ts used for easmunicatlon 
between experimenter and subject, a dedL of y x 8" cards containing instructional items^ 
and four sheets ("panels") with infoxnation relating to the Instructional, itsms. The 
wooden screen had a window in its center In which the eagperlmeater placed cards so the 
instructional items could be seen by the subject. A holder on the experimenter's side 
of the window supported the cards, and also prevented the subject from observing the 
experimenter through the window at any time. The information sheets were attached to 
the subject's side of the screen for all experimental training sessions, and could be 
used by the subject as an aid in answering the instructional items; subjects could not 
use the sheets while taking the criterion qiois, however. 

ladh of the six sets of experimental equipment could be operated under any one of eight 
teaching procedures. These eiijit teaching procedures represent the eic^t ooBblaatioas 
of three experimental variables, each variable having two possible values as follows: 

(l) Besponse mode (multiple ehoice versus constructed response); (2) 8ise of steps 
between successive items to be tsu^ (smi^ll steps versus lurge steps); and (3) 
of it«i sequance control (branching versus non-branching). Thus each subject was run 
under one of the foUowing conditions: 

a. Nultlple Choice; aMO.1 Steps; Bo Branching 

b. Oonstructed Response; Shall Steps; Bo Branching 

c. Multiple Choice; Large Steps; Bb Branching 

d. Constructed Response; Large Steps; Bo Branching 

e. Multiple Choice; Shall Steps; Branching 

f. Constructed Response; Small Steps; Branching 

g. Multiple Choice; Large Steps; Branching 

h. Constructed Response; Large Steps; Branching 
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Subiects and experimenters vere randomly assigned to the eight expei^ental conditions, 
vitbln the restriction .hat ten subjects were trained under each condition. 

Printed instructions (See Appendix I, II ) were read aloud ^ 

-tart of the training session. These instructions described the task to be perform^ 
by the subject, and informed him that both accuracy and speed of learning were importan . 

^NgTRUCTED RESPONSE VS. MJITIPLE CHOICE OPERATION^ In the constructed ^^^poMe^e of 

operation there are tw pushbuttons on the subject's side o . ® screen 
lights on the experimenter's side. The experimenter places in the 

a ^d containing a constructed response itemo This item consists of a state^t with 
or^ OT ^re bS^ spaces representing words to be filled in by the subject. T^e subject 

on a slip of paper and then tape Ughtly on the screen to l>^»te 

that he has made his response. Upon hearing the tap the 

window a cardboard mask, revealing to the subject the correct raswer ..o the Itra. The 
subject compares his own answer against the correct answer, 
pushbuttons, marked "right" and "wrong" respectively, to actlTOte toe 
light on the experimenter's side. This indicates to the ejqperi^yer OT not 

the subject got toe item correct.* • The experimenter removes this card and goes on 
to the next card, continuing until he has finished the He then ^ 

entire process, but this time uses only the items missed by the object “e 
trial. The procedure is continued through successive trials until the subject has 

answered every item correctly. 



In the multiple choice mode there are five pushbuttons on the subject s “We of toe 
screen, activating five lights on the e:Q>sniiienter ' s side. These 

by the subject to indicate his answer to toe multiple choice it^ prese^ed to . 

In addition there are two pushbuttons used by the experimenter to activato llpts on 
the subject's side, telling the subject whether he has selected a correct or incowect 
response. Multiple choice items consist of statements with blank spaces repress ng 
missing words. Below the statement are listed two to five possible ctoUes, on^f 
todch is to he selected by the subject to answer the item. This syertlon ^ 
pushed by the subject's pressing the pushbutton having the same letter lahy M the 
Lswer which he wishes to indicate. The various choices are presented to tto s““^ot 
simultaneously with the item, there being no mask of ty f 

response items. The subject must continue to respond to ^ in 

the correct answer, upon toich the experimenter replaces that item with the , 

toe deck. After he has completed the first trial (e.g., the firs r<m «niv the " 

the experimenter continues with additional trials as required, each t^ using only the 
cards that the subject missed in his first response on toe preceding trial. 

"SMALL STEPS" VS. "LARGE STEPS." Sope subjects were trained with 

containing many ( 104 ) items. ®ese decks are called "small step" decks, 

supposedly require Uttle effort on the part of the subject to answer a partlcul m U em 

once he has gone through the preceding items. Other subjects were trained wUh 

step" decks containing fewer ( 56 ) items, and ostensibly requiring more effort to answer 

successive items. 

* An informal check was made to determine the dependability of the subjects’ evaluation 
of their own responses. Several randomly selected subjects were observed during train- 
ing, and after training their slips were compared with toe correct answers. Resul s 
of toe check indicated that subjects were generally accurate in their evaluations. 
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Prom the standpoint of the operation of the teaching machines, there are no proc^ural 
differences between "small step" decks and "large step" decks, except in the number of 
items presented to the subject. 

"HRAHCHIHG" V8. "NON-BRANCHHIG." This variable determines the sequence in which items 
are prese^^ to the subject. ~~In the "non-branching" mode, the experiment. ’ simply goes 
throu^ aLl the cards of the deck in order. In the "branching" mode, certain items are 
removed from the deck if the subject answers certain other items correctly on the first 
try. The items to be removed are taken out of the deck by a second experimenter, so 
they are never presented to the subject. Thus, in the "branching" mode, the exact 
milder and sequence of items are not fixed^ but depend on the perfoimance of the subject. 
A more complete description of the criteria for omitting items is given in the following 
section on Subject Material. 

PEBFORMAKCE MEiUSURES. The following performance measures were recorded for each subject, 
regardless of mode of Instruction: 



SUBJECT MATERIAL 

The eight item sets (decks) and the three instructional sheets used in the experiment 
were based on a portion of a college course in elementary psychology used at Harvard 
University. The first item set obtained was the constructed response, small step deck, 
without branching. This set consisted of the first lOk items from a larger series of 
items developed at the Harvard University Psychological Laboratory. Samples of the 
Items are shown in Appendix VI. The remal ning seven sets represent c ombina tions of 
the following transformations of the original Items: 



(1) A number of possible alternative answers were provided for each 
Item, to make the multiple choice decks. 

(2) Certain Items were eliminated from the original set of 104 Items, 

to make the "large step" decks of 56 items. The items eliminated 
were Judged to be largely redundant, in the sense that they related 
to concepts cQready covered by other items, and added little extra 
information. ^ 

(3) To the "branching" decks, coded Instructions for the brandling 

procedure were marked on the backs of certain cards. These instruc- 
tions Indicated yhlcih cards were to be eliminated from the deck dur- 
ing the experiment If the subject correctly answered the marked 
cards. B*or convenience, a marked card will be referred to as a 
"branch card"; the cards to be eliminated ^en a 'Tiranch card" Is 
correctly answered will be called "conditional skip" cards. In 
general, a "branch card" was the first of several Items in the deck 
covering a particular concept, while the associated "conditional 
skip" cards consisted of the remaining Items covering essentially 
the same concept. "Small step" branching decks included 13 "branch 
cards" and 39 "conditional skip" cards, while "large step" branching 
decks Included 12 "branch cards" and 22 "conditional skip cards. 



(1) Time required to finish tr ain i n gs 

( 2 ) Score on multiple choice portion of criterion test. 

( 3 ) Score on constructed response portion of criterion test. 
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The three instructional sheets, or "panels," used in the study 
covered most of the major concepts presented in the instructional 
items, hut in the form of definitions and brief descriptive 
materials. Appendix V contains several examples of statements 
from the panels, vhich were prepared by Harvard University. 



CRITERION TEST 

The criterion test consisted of 36 questions, of which I 9 were constructed response 
(fill-in) and 17 were multiple choice. All of the questions were based upon the 
material contained in the origineJ. (constructed response; small step; without branching) 
set of instructional items. The person who prepared the criterion iteias had not seen 
the othe* jeven item sets prepared for instruction; conversely, the persons preparing 
the alternate item sets had not seen the criterion questions. 

None of the criterion questions were duplicates of instructional items, though many of 
the same words were used as correct responses . Most of the criterion questions were of 
the application type, in which a situation is presented and the respondent identifies 
the principle involved or atten 5 )ts to explain or predict an outcome. Examples of 
criterion test questions are shown in Appendix IV. 

Reliability of the criterion test, as estimated by the application of Kuder-Richardson 
Formula 20 to the scores of the experimental subjects, was as follows: whole test, .89; 

nstructed resixDnse portion, . 85 ; multiple -choice portion, .79* Test-retest correla- 
tions for experimental subjects were .8l for the whole test, .79 for bhe constructed 
response portion, and .58 for the multiple choice portion. 



EXPERI 



TAL SUBJECTS 



Tbe 80 experimental subjects euid the 104 members of the control group were taken from 
beginning psychology classes in Santa Monica City College. They had been in the 
psychology course for about one month, using F. Ruch's Psychology aiid life cub a text 
and covering some general background in the area of psychology, but had not been given 
specific instruction in any material closely related to the subject material used in 
the experiment. 



RESULTS 



1, A psychology pre-test (See Appendix III) was given to all experimental 

subjects and also to the students who had not been selected for the experiment. Means 
and SB's for the two groups are given in Table I. As shown in Table II, no significant 
difference was obt.ained between the 8C' ejqperimental S*s and the 104 non-experimental 
S' 6 using a "t" test. The highest possible score on the pre-test was 6 I. 

After the experimental training period a criterion test, covering applica- 
tions of the concepts that were being taught, was administered to experimental and non- 
experimental groups. The criterion test consisted of two subtests, one with multiple 
choice items, the other with constructed response (fill in) items. 
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Table I gives the mean scores and SD’s for the eiqperimental and control 
groups. Maximuni possible scores on the criterion were 19 and 17 for the constructed 
response and multiple choice subtests respectively. 

The experimental, group was superior to the control group on the total 
criterion on both the multiple choice and constructed response subtests, as ^own 
in Table II. Differences were significant at the .01 level. 



3. The criterion test was re -administered to the experimental S's three 

weeks after the first administration as a meac\u:e of retention. Bfeans and SB's for 
the experimental group retest are given in Table I. As shown in Table H, no signif- 
icant difference was obtained on the total criterion (multiple choice plus con- 
structed resx>onse), \dien the mean of the first administration was comp€ured with the 
mean of the second administration. On the multiple choice criterion subtest, however, 
retest scores were significantly higher (.01 level) than original test scores. This 
improvement in performance may be attributable to the fact that students became 
ac<iu8Linted with the criterion questions, although it is difficult to ejqplain why a 
similar Improvement did not occur on the constructed response subtest. 



The evidence from these first three analyses indicates that the automated teaching 
materials used with the eiqperlmental subjects resulted in significant lea rn i n g of the 
concepts taught and that this learning is retained at least for a three week period. 
These findings do not mean that the experimental S's exceeded groups receiving con- 
ventional cl8tfisroom Instruction on the same materisLls, since the control S's were 
not being taughb the same concepts as the e 35 >erimental group. Hie purpose of the 
experimental versus control group conparison was to determine whether any learning 
was taking place in the experimental group over that which mi^t have occurred 
without such training. 
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TABLE I 



MEANS AND STANDARD DEVIAnONS FOR EXEERIMESTAL AND CONTROL GROUPS 
ON Hffi-TESr AND GRTCffilON SUBTESTS 



SOURCE OF SCCNtES 




1 Experimental Group 


Control Group 




Mean 


SD 


Mean 


SD 


PRE-TEST 


** 5.5 


8.0 


kk.k 


7.7 


FIRST 

AINUNISIRATIQN 
OF CRITERION 
TEST 


Miiltiple 

Choice 


11.6 


2.5 


10.5 


2.6 


Constructed Response 


13.5 


3.9 


8.6 


3.6 


Total Criterion 


25.1 


5.9 


19.1 


5.2 ' 


SECOND 

AlMLNlinRAnON 
OF CPJTERION 
TEST 


Moltiple 

Choice 


i 

* 

12.h 


2.5 


NOT I 


Constructed 

Response 


1 

j 13 .*t 


3 .^ 




TERED j 

! 

1 

i 

i 

i 


Total 

Criterion 


1 

1 25.8 


5.3 


1 




-y 




• 

ir 
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TABLE 


II 








COMPARISONS FOR EXPERIMENTAL AND CONTROL GROUPS 
ON IRE'TEST AND CRITERION TEST 


• 




COMPARISON 


t 


df 


P 




EXPERIMENTAL GROUP MINUS 
CONTROL GROUP (HI PRE-TEST 


1 1.00 


102 


not slgnlf. 


( 


SZFfitIMEHTAL GROUP 
MINUS OON 3 ROL OlOUP 
ON ORIGINAL CRITERION 


Multiple 

(Riolce 

Subtest 


2.99 


195 


.01 




TEST 


Constructed 

Response 

Subtest 


9.07 


195 


.01 






Tbtal 

(pterion 


7 35 


195 


.01 




ORIGINAL CRITERION 
TEST MINUS RETEST 
FOR EXPSOMERTAL 
GROUP 


Multiple 

(Riolce 

Subtest 


2.92 


70 


.01 




Constructed 

Response 

Subtest 


. 6 k 


70 


not slgnlf. 






Total 

Criterion 


1.67 


70 


not slgnlf. 


o 
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Table III contains pre-test scores and criterion test scores for the eight 
experimental treatment groups . The correlations between pre-test scores and criterion 
scores were as follows: 

Correlation between pre-test and constructed response scores = .Ull 
( -01 level of significance) 

Correlation between pre-test and multiple choice scores - .kok 
(.01 level) 

Correlation between pre-test and total criterion = ,hk2 (.01 level) 



j. Kcbulto of cc/8o:lance analysis of the constructed response criterion 

measures for the eight experimental groups are shown in Table IV. Pre-test scores 
were used as a control variable in this emalysis because of the significance of the 
correlation between pre-test and constructed resjonse criterion scores. A significant 
main effect was found for '^''e size of step factor (.05 level of significance ), in the 
direction of higher scores for small step trainees. A significant interaction was 
also found between the mode of resjonse emd branching variables (.05 level). !l^e 
assumption of homogeneity of regression was tested and found to be satisfied. 

Table V gives observed mean constructed resxx>nse scores for comparison of 
the effects of the three experimental variables (response mode, step size and branch- 
ing procedure). Mean constructed response criterion scores are e^.so given for the 
four combinations of response mode and branching procedure, since this interaction 
vas found significant in the covariance analysis. 

The right-hand column of Table V shows the constructed response criterion 
means adjusted for pre-test scores. 

6. Covariance analysis of the multiple choice criterion measures of the sub- 

jects in the eight experimental groups was also performed wi.th psychology pre-test 
scores as a control variable. The results of this analysis, summarized in Table VI, 
yielded no significant main effects or interactions among the experimental groups. 

Table VII shows the observed and adjusted multiple choice criterion scores for the 
main treatments. 

7- Covariance analysis of the total criterion measures (multiple choice plus 

constructed response), using the psychology pre-test as a control veiriable, is summar- 
ized in Table VIII. No significeuit main effects or interactions were obtained among 
the experimental groups. Observed and adjusted mean scores for the experimental groups 
on the total criterion test are shown in Table IX. 

8. An analysis of variance of differences among the eight experimental groups 

with respect to the amount of time taken to con 5 >lete training was also performed. All 
three main effects are significant at the .01 level. The constructed response training 
condition took more time them the multiple choice condition; the amed.1 step condition 
took more time than the large step condition; and the non-bremching condition took more 
time than the branching condition. Interactions were not significant. The mean scores 
on the time criterion eo‘e shown in Table X end the analysis of vcLriance is sunmeurized 
in Table XI. 



f 
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TABLE III 



NBAH SCORES FOR EIGHT MLPBtlMHyrAL GRO UPS O W IBB ~T E 8 T 
AHD OH MLTI 3 ?IPLE CHOICE AHD C 0 H 8 TRUCTED RESP 0 H 8 B CRITBRIA 







- II 


TRSATMEffT GROUPS 


NEAH OH 


OBSmED NEAH GRITEE 


tlOH SCCR^ 


Response F 


step 




FRE- 


MC 


CR 


TOTAL 


Node 


Size 


Branching ' 


TEST 








1 . CR 


Sisal 1 


Branch 


i* 9 .^ 


12.1 


13.8 


25.9 


2 . MC 


SNall 


Branch 


45.0 


12.1 


i4.3 


26.4 ^ 


3 . CR 


LsrgiK 


Branch 


45.0 




13.3 


25.2 


4 . MC 


Large 


Branch 




10.6 


13.2 


23.8 


5 . CR 


s»ai 


lb Branch 


kl.2 


11.6 


15.1 


26.7 


6 . NC 


SNaU 


Ho Branch 


i^ 5.3 


11.9 


13.6 


25.6 


T. CR 


Large 


Ho Branch 




12.2 


lU.O 


26.2 


P, NC 


Large 


Ho Branch 


^5.3 


10.6 


10.7 


21.3 1 


ALL 

SR 0 UP 8 

U — = 






^ 3-3 


11.6 


13.5 


25.1 
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TABLE IV 



COVARIANCE ANALYSIS OF CONSTRUCTED RESPONSE CRITERION SCORES 
USING A PSYCHOLOGY HUB-TEST AS A CONTROL 



SOURCE OF 
VARIATION 



ADJUSTED SUM 
OF SQUARES 



df 



ms 



A. Mode of 
Response 


29.22 


B. Size of 
Step 


46.99 


C. Branching 


1.27 


AB 


11.05 


AC 


53.28 


BC 


18.34 


ABC 


2.23 


Within Cells 


846.88 


TOTAL 


1009.26 



71 

78 



29.22 



2.48 



46.99 


3.99 ♦ 


1.27 


.11 


11.05 


.94 


53.28 


4.52 ♦ 


18.34 


1.56 


2.23 


.19 


11.93 





♦ .05 Level 
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TABLE V 



OBSERVED AND ADJUSTED MEAN SCORES OH COHSTRUCTED RESPOHSE CRITERIOH 

SUBTEST 



GROUPS 

(Training Node) 


OBSERVED MEANS OH 
(XHVSORUCTED RESPOHSE 
CRITERIOH 


COHSTRUCTED RESPOHSE 
MEANS ADJUSTED FOR . 
PRE-TEST SCORES 


Constructed Response 


14.05 


14.14 


Multiple Choice 


12.95 


12.86 


Snail Step 


14.20 


i4.31 


Large Step 


12.80 


12.69 


Branching 


13.65 


13.21 


Hb -Branching 


13.35 


13.79 


Branching with 
Constructed Response 


13.55 


12.93 


Branching vith 
(tultlple Choice 


13.75 


13.50 


Ho -Branching with 
Constructed Response 


li^.55 


15.35 


fo -Branching vlth 
Multiple Choice 


12.15 


12.92 






ERIC 
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TABLE VI 

CX5VARIAHCE ANALYSIS OP lULTIPLE CHOICE CRITERION SCORES 
USING A PSYCHOLOGY PRE-TEST AS A CONTROL 











SOURCE OP 
VARIATION 


ADJUSTED SUM 
OP SQUARES 


df 


ms 


P 


A. Mode of 
Response 


10.22 


1 


10.22 


1.94 




B. Size of 
Step 


9.20 


1 


9.20 


1.74 




C. Branching 


.99 


1 


.99 


.19 




AB 


15.66 


1 


15.66 


2.97 




AC 


.82 


1 


.82 


.16 


• 


BC 


.03 


1 


.03 


.01 




ABC 


1.03 


1 


1.03 


.20 




Nlthln Cells 


37^.81 


71 


5.28 






TOTAL 


412.76 


78 









ERIC 



- 17 - 

TABLE VII 

OBSERVED AMD ADJUSTED MEAN SCORES OH MJLTIPLE CHOICE CRITERIOH 

SUBTEST 



GROUPS 

(Training Mode) 


OBSERVED MEANS ON 
MJLTIPLE CHOICE 
CRITERION 


- — : 

MJITIFIE CHOICE MEANS 
ADJUSTED FOR IRE-TEST 
SCORES 


Constructed Resx)on8e 


11-95 


11.97 


Kultlple Choice 


11.30 


11.3^ 


Small Step 


U.93 


11.97 

1 


Large Step 


11.33 


I 

U.89 


branching 


11.68 


11.66 

! 


So Branching 


11.58 


U.6O 
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TABLE VIII 



CX)VARIANCE ANALYSIS OF TOTAL CRITERION SCX 5 RES USING A 
PSYCHOLOGY PRE-TEST AS A CONTROL VARIABI£ 



SOURCE OP 
VARIATION 


ADJUSTED SUM 
OF SQUARES 


df 


ms 


F 


A. Mode of 
Response 


72.15 


1 


72.15 


2.65 


B. Size of 
Step 


99.96 


1 


99.96 


3.68 


C. Branching 


It. 65 


1 


4.65 


.17 


AB 


5U.73 


1 


5 lt .73 


2:01 


AC 


65. 8 U 


1 


65.8lt 


2.^2 


BC 


18.35 


1 


18.35 


.67 


ABC 


5.57 


1 


5.57 


.20 


Within Cells 


1930.80 


71 


27.19 




TOTAL 


2252.05 


T 8 







i 



o 

ERIC 
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TABLE DC 

OBSERVED AND ADJUSTED MEAN SCORES ON TOTAL CRITERION 

TEST 



GROUPS 

(Tralnlog Mode) 


OBSERVED MEANS ON 
TOTAL CRITERION 


TOTAL CRITERION MEANS 
ADJUSTED FOR PRE-TEST 
SCORES 


Constructed Response 


26.00 


26.02 


iultlple Choice 


24.28 


24.24 


Small Step 


26.15 


26.19 


Large Step 


24.13 


24.09 


Branching 


25.32 


25.30 








fo Branching 


24.95 

1 - i 


24.97 
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TABLE X 

MEAN SCORES ON 1BAINIHG TIME CRITERION 



GROUPS 


MINUTES 


Constructed Response 


54.4 


Multiple Choice 


44.4 


Small Step 


57.3 


Large Step 


41.5 


Branching 


43.8 


No Branching 


55.0 
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TABLE XI 

AMALYSIS OF VARIAHCE OF TIME REQU] 

TRAHUBO 






TO COMPLETE 






SOURCE OF VARIATION 


88 


df 


ais 


F 


K (Mode of Response) 


2000.00 


1 


200c. 00 


11.66 ** 
% 


3 (Site of Step) 


4992.80 


1 


4992,80 


29.11 


^ (BMiCblng) 


2508.80 


1 


2508.80 


l4.62 ** 




42.05 


1 


42.05 


.25 


KC 


1 

48.05 


1 


48.05 


.28 


lUBC 


540. 80 


1 


540.80 


3.15 


fitbln Cells 


12347.80 


72 


171.50 




POTAL 


22642.75 


79 







* * .01 Level 
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DISCUSSION 



CONgTRUCTED RESPONSE CRITERION DATA 

SI2E OF STEP VARIABIE: The finding that treatment groups given more items 

( gmfti 1 steps ) 1 earned more then groups receiving few items is in accord with 
B. F. Skinner’s emphasis on the importance of small steps in writing instruc- 
tional item sets. The superiority of small steps in total amount learned 
must be weighed, however, against the significantly greater trai nin g time 
required by small step trainees than by large step trainees. The best 
choice of item step size for any applied teaciiing situation will be deter- 
mined in part by the extent to which required training time is a critical 
consideration . 

BRANCHING VARIABLE: In the comparison of large item steps and sm€tll steps it 

was found that step trainees, who were given more items, learned more 

than large step trainees. In view of this finding it is notable that branching, 
with its consequent reduction in total number of items presented, did not differ 
significemtly from non-branching in amount taught. One possible explanation 
lies in the fact that the '’Ifference in the number of items given to branching 
and non-branching trainees was* less than the difference between large and small 
step trainees. A more important factor, however, may have been the way in which 
items were eliminated in the branching procedure. An item was skipped only 
after the trainee had demonstrated some knowledge of the concept tau^t by that 
item, with the result that every major concept was covered by at least one item. 

At the same time, the branching procedure made possible a significant decrease 
in required training time, as conq)ared with the non -branching procedure. 

When both the amount learned and the required training time are considered, the 
branching procedure api>ears to offer an over-eLLl advantage over non-branching. 

The present investigators feel, moreover, that branching, or other types of 
control flexibility, offer even greater potential for future improvement teach- 
ing machine effectiveness In the study reported here, limitations due to human 
simulation of the teaching machine control mechanism placed heavy restrictions 
on the branching procedure. Bie greatly simplified skipping procedure resulted ^ 
in a significant advantage in trsdning time but not in amount learned. More 
complex forms of control flexibility, using some type of automatic equipment, 
may prove superior to pre -determined sequence control in both amount learned 
and speed of learning. 

RESPONSE MODE VARIABIE: Under the conditions of the experiment reported here, 

student resjxjnse mode did not significantly eiffect the amount learned by the 
students. Since required training time was significantly less for multiple 
choice trainees than for constructed response trainees, the over-all advantage 
appeeurs to be with the multiple choice mode. It is possible, of course, that 
the results would be different under other experirjntal conditions, and further 
research in the area may prove valuable. 

An iraportaat consideration in the eveiluation of response mode for applied pur- 
poses is the ease with which a particular mode can be incorporated into an 
actual teaching machi-.e. This practicsLl consideration must be weighed along 




A. 
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with the teaching effectiveness of the response mode. The use of the multipit 
choice mode, rather than the constructed response mode, greatly simplifies the 
design of any teaching device \^iich is intended to evaluate student responset 
autODiat i cally . 

pTggpONSE MODE-BRAWCHIHG IHTEEtACTIOH: The significant interaction between 

response mode and branching procedure can probably be attributed to the 
combined effects of two factors, neither of which is sufficiently powerful 
to have statist! csill^/ significant effect when taken separately. The first 
factor contributing to the interaction is the number of items presented to 
the subjects. By this hypothesis the constructed response trainees learned 
more without branching than with branching because the branching procedure 
caused a reduction in the total number of items. The second factor contrib- 
uting to the interaction effect was the relation between the response mode 
required in the training session, and that required in the criterion test. 

As previously noted, all the criterion scores being compared here were ob- 
tained from the constructed response portion of the criterion test. This ml^t 
Irell be eJtpected to give an advantage to the constructed response trainees due 
to generalization decrement on the part of the multiple choice trainees. 

Table V shows that main effe t differences in favor of the non-branching and 
constructed response conditions, though non- significant, tend by their 
direction to support the "combined effects" interpretation presented here. 



MULTIF1£ CHOICE CRITERION DATA 



Although the mean of the eciqperiinental S's on the multiple choice criterion 
significantly exceeded the control group mean, no significant differences 
were obtained among the different experimental groups. Differential effects 
of the various experimental treatment combinations on the multiple choice 
criterion appear to have been masked by a larger effect connoon to all 
experiment6^. groups. This m«j»klng error variance can probably be attributed 
to the nature of the multiple choice criterion subtest. Of the 17 multiple 
choice items, nine had four alternatives, two had three eO-ternatives, and 
six had only two alternatives. The probability of obtaining correct answers 
on a chance bsisls could account on the average for approximately one third 
of the items. The largest portion of the variance of the scores on the 
multiple choice test may be due to this chance factor. Students of the 
sort used in the study are usually highly skilled in picking out subtle 
specific determiners by which they can eliminate implausible alternatives 
on grounds other than factual content. These skills, which together with 
day-to-day fluctuations in individual performance are treated as systematic 
variance in the Kuder -Richardson method of estimating reliability, contribute 
to over-estimation of the effective reliability of the test, in this case, 
the Kuder-Richardson reliability estimate was . 79 * A more realistic estimate 
of reliability is given by the multiple choice criterion test-retest correla- 
tion, which was only .58. 
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An alternative to the "masking” interpretation is that there was actually ^ 

no difference in effectiveness among the experimental treatments. The 
results on the constructed response criterion and the operational differ- 
ences among the treatments, however, tend to place doubt on this second 
hypothesis. Althou^ not significant, the direction of differences among 
Lhe of the experimental groups shown in Table VII tend to support 

the results obtained on the constructed response criterion. 



TOTAL CRITERION TEST DATA 



The results of the analysis of the total criterion follow from the results 
obtained on the analyses of the two subtests. Error variance in the 
multiple choice portion of the total criterion appears to have masked 
inter -group differences so that these differences are not significant. 

It can be noted in Table VIII that the covariance F test approaches but 
does not reach significance at the .05 level for both the size of the 
step factor and the mode of response - branching interaction. The 
direction of the means in Table IX also agrees with differences obtained 
on the analysis of scores on the constructiid response criterion. 



TIME CRITERION 



Differences in required training time cure generally in the expected 
direction. Small step groups and non-branching groups took longer than 
large step and branching groups because they were required to answer 
more questions. The fact that constructed response groups took longer 
than multiple choice groups indicates that the composition and writi^ 
of answers is more time consuming than the recognition of a "correct" 
solution Mw^T^g several alternatives, even though multiple choice 
"trad-uees were sometimes required to make several selections before 
choosing the right answer. 




SUMMARY 



Eight groups of ten Junior college students were given an erperiaental training 
session with manually simulated teaching machines, each group being tau^t with 
a different mode of teaching machine operation. The three independent variables 
were student response mode, size of steps between successive items and sequencing 
(branching) procedure. A written criterion test was given to all subjects 
iMediatSy after the training session, and again thiee weeks later. The same 
criterion test was given to a control group which had no training with the 
teaching machine, but which came from the same school classes as the experimental 
subjects. Dependent variables were the required teaching machine training time 
and scores on the criterion test. Scores on a pre-test were used as a control 
variable in a covariance design. Major results were as follows: 



(1) Use of the simulated teaching macbines led to significant 
learning by the subjects, as determined by comparison with 
the control group. 

(2) The multiple choice response mode took significantly less time 
than the constructed response mode. No significant difference 
was obtained between response modes on the criterion test. 

( 3 ) RmRii item steps required significantly more training time, 
but also yielded significantly higher test scores than large 
item steps on the constructed x^esponse criterion subtest. 

(4) The branching conditions required less training time than non- 
brauaching, but. were not signi f ieeuitly different on the criterion 
test. A significant interaction was obtained between the mode 
of response and branching variables or* the constructed resxonse 
criterion. This interaction resulted from a hi^ mean criterion 
score obtained by the constructed response - non-branching group. 

( 5 ) No significant differences were obtained among the experimental 
groups on the multiple choice criterion subtest, or on the total 
(multiple choice plus constructed response) criterion test. 
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APPENDIX I 



DIRECTIONS TOR SUBJECT - OONSDIUCTED RESPONSE ITEMS 



The object of this experiment is not to find out hov much you 
know, or to compare you with any other student, but to find out vhat 
teaching procedure does the best Job of teaching you something about 
psychology. 



You will be shown a series of questions. Each question will 
contain a statement with one or more blank spaces. You are to write 
your answers on a sheet of blank paper pad in front of you. As soon 
as you have written your answers you tap the screen with your pencil, 
indicating that you are ready to see the correct answer. The correct 
answer will then be revealed in the window beneath the original ques- 
tion. If your answer was correct, you press the pushbutton narked 
”ricpit." If your auiswer was wrong, you press the "wrong" button. 

Hold the "right" or "wrong" button down until the next card appears. 
Then tear off the sheet with your answer on it, and drop it in the 
box in front of you. You are then ready to go through the answering 
procedure with the next card Just as you did with the first card. 



It is up to you to decide whether you have answered a particu- 
lar question correctly. If a card has more than one blank, you must 
answer 6dl parts correctly or the whole question must be coimted as 
"wrong." In OxXler to be marked "right," your answer must use exactly 
the same word or words as the correct emswer, or words that mean the 
sane thing. 



A record will be kept of how much you learn, and of the time 
required for you to learn it. It will be up to you to decide how much 
time you should spend in trying to answer a particular question 
correctly. 



er|c 
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APPENDIX II 



DIRECTIONS FOR SUBJECT - MULTIPLE CHOICE ITEMS 



The object of this experiment is not to find out how much you 
know or to compare you with any other students, but to find out what 
teaching procedure does the best Job of teaching you something about 
psychology. 

You will be shown a series of multiple choice questions. Read 
the question, then indicate yo\ir euiswer by pressing the pushbutton 
corresponding to your choice. For example, if the euiswer you choose 
is alternative B, press pushbutton B until you see either a red light 
or a green light. The green light means that you were correct. The 
red light means that you were wrong and you must try again until you 
get it rif^t. This samo proced’ire will be used for the next card and 
the next, until you have finished. 



A record will be kept of how much you learn, and of the time 
required for you to learn it. It will be up to you to decide how much 
time you should spend in trying to answer a particular question 
correctly. 




* 



m 
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APPKHDIX III ^ 

EXAMPLES OF IRE-TEST QUESTIOHS 



1. The humoral theory va» proposed by: (a) Kretschmer; (b) Mesmer; 

(c) Hippocrates; (d) Sheldon. 

2. The oldest known characterological theory is: (a) phrenology; 

(b) humoral theory; (c) somatotype theory; (d) psychophysics. 

3. Sheldon is chiefly known for his connection wit**: (a) the soma- 
totype theory; (b) the humoral theory; (c) phrenology; 

(d) psychophysics. 

4. A person who believed in phrenology would attempt to leam about 

I>ersonalty traits by studying: (a) body fluids; (b) body build; 

(c) reactions to specific stimuli; (d) the shape of the skull. 

5. Which of the following would be most likely to fluctuate between 

iBOods of happiness and depression: (a) schlzothyndc tyx>e; 

(b) leptosome; (c) Mesomorph; (d) cyclothymic type. 

6. Which of the following would probfiO)iy be most interested in 
athletics and other muscular activities: (a) endomorph; 

(b) mesomorph; (c) leptosome; (d) pyknic type. 

7. An imbalance of magnetic fluids in the body was regarded as the 

cause of illness by: (a) Mesmer; (b) Hippocrates; (c) Kretschmer 

(d) Fechner. 

8. Mesmerism led to the development of the technlq.ue known as: 

(a) projective testing; (b) shock therapy; (c) hypnosis; 

(d) the c31nlcal method. 

9. Ihllippe Pinel is noted primarily for his work in: (a) experimental 
physiology; (b) the mind-body problem; (c) psychophn^lcs; 

(d) the refom of mentSLl hospitals. 

10. The father of psychophysics is: (a) Mesmer; (b) Final; 

(c) Helmholtz; (d) Fechner. 



o 






APPEaJDIX IV 



EXAMPLES OF CRITERION QUESTIONS 



Directions: On the multiple choice items, put a check (X) on the line to 

the left of the correct answer. On the Incomplete sentences, 
print the correct answer in the space provided. 



1. Bill, a candidate for the draft, was tapped on the knee with a rubber 
hammer; his leg Jerked. After waiting five minutes the doctor tapped 
his knee again, but harder. The movement of his leg the second time 
should be: 

a) greater 

b) the same 

^c) smaller 

2. Which statement is correct? 

^a) The word "reflex" means the same as"response. " 

b) Any reflex action is a response to a stimulus. 

c) Any response to a stimulus is a reflex action. 

d) None of the above stataaents is correct. 

3 . Which of the following is a stimulus? 

e.) Saliva in your mouth during a meal. 

b) The sound of a pencil hitting the floor. 

c) Picking up a pencil that has fallen. 

d) A sound that is not heard. 



h. When a loud noise occurs near an infant he will normally Jerk his 

body. This combination is called the "startle . " 

5* A reflex response always comes ^the stimulus. 

6. When a conditioned stimulus precedes a response we say that the 

stimulus the response . 



APPENDIX V 



EXAMPLES OF INSroUCTIONAL PANELS 



A light (a stimulus ) flashed into the eye causes the pupil to 
contract ( a resx>onse ). This relation between the stimulus amd 
its response is called the pupillary reflex . Tlie time elapsing 
between the onset of the stimulus and the onset of the response 
is the latency of the reflex. The lowest intensity of light 
which will elicit contraction Is called the threshold of the 
stimulus . 



MEM TECHNICAL VTOKDS 

stimulus 

response 

i;eflex 

latency 

elicit 



threshold 
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APPENDIX VI 



examples op traihiiig items 



3. 

if. 

5 . 

6 . 



8 . 



9. 



A doctor taps your knee (patellar tendon) vlth a rubber hamner to test your 

ti "reflexes 



If your reflexes are normal you_ 
with a knee Jerk. 



to the tap on the knee 
" respond" ( or react ) 



In the knee Jerk reflex, the knee Jeikls the_ 



response (or reaction} 



The stimulus which elicits the knee Jerk response is the ^ — — , — — * 

tap (on the knee) 

To explain the knee Jerk you need only describe the^ — — — .. — * 

^ stimulus (or tap on the knee) 



The tap on the knee_ 



the knee Jerk response. _ ^ 

elicits (or causes, produces, brings about) 



In the knee Jerk reflex the temporal order is first the (l) ^ 

4 .v,^„ 4 -Ka. (o\ (l) stimulus (or tap) 

then the (2) response knee Jerk) 

The interval of time ahieh necessarily elapses between the stimulus 
response is called the latency. The time between tapping the knee and the 

knee Jerk is the of the knee Jerk reflex. 

— latency 

The minimum intensity of a stimulus Just sufficient to elicit a respoMe is 
called the threshold of the stimulus. A tap on the knee will elicit the 
knee Jerk only if the tap exceeds the 

threshold 



10 . 



A very hard tap on the knee elicits a(n) (l) . . ; t 

while a tap barely above the threshold elicits a(n) (2) 



knee Jerit 



(1) large (or big, intens^ 

(2) small (or little, feeble) 



o 
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